Papers with image representations
Coherent and Concise Radiology Report Generation via Context Specific Image Representations and Orthogonal Sentence States (2021.naacl-industry)
Copied to clipboard
| Challenge: | Neural models for text generation are often designed in an end-to-end fashion, limiting their practical usability in downstream applications. |
| Approach: | They propose a method to compute image representations specific to each sentential context and exploiting diverse sentence states to ensure topical continuity and content diversity of generated radiology reports. |
| Outcome: | The proposed method outperforms baselines on objective metrics and human evaluations by 18% and 29% respectively in the evaluation for informativeness and content ordering respectively. |
Prompt Refinement with Image Pivot for Text-to-Image Generation (2024.acl-long)
Copied to clipboard
| Challenge: | Recent advances in text-to-image generation have markedly expanded the boundaries of digital artistry, enabling the creation of visually compelling images with unprecedented ease. |
| Approach: | They propose to decompose the prompt refinement process into two tasks: inferring user-preferred images from user languages and translating them into system languages. |
| Outcome: | Experiments show that PRIP outperforms baselines and transfers to unseen systems in a zero-shot manner. |
Improving Few-Shot Image Classification Using Machine- and User-Generated Natural Language Descriptions (2022.findings-naacl)
Copied to clipboard
| Challenge: | Humans can efficiently learn about new concepts from language descriptions, and we propose a new machine learning model, LIDE, which has a text decoder to generate the descriptions and a decoded text encoder to obtain the text representations of machine-generated descriptions. |
| Approach: | They propose a model with a text decoder to generate the descriptions and a corresponding text encoder to obtain the text representations of machine- or user-generated descriptions. |
| Outcome: | The proposed model outperforms baseline models with machine-generated descriptions and with high-quality user-generated models with high quality explanations. |
Describing Images Fast and Slow: Quantifying and Predicting the Variation in Human Signals during Visuo-Linguistic Processes (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing models of visuo-linguistic variation are weak to moderately trained to capture such a variation in visual outputs. |
| Approach: | They use a corpus of Dutch image descriptions with eye-tracking data to investigate the nature of the variation in visuo-linguistic signals. |
| Outcome: | The proposed model lacks biases about what makes a stimulus complex for humans and what leads to variations in human outputs. |
Uncovering the Full Potential of Visual Grounding Methods in VQA (2024.acl-long)
Copied to clipboard
| Challenge: | Visual Grounding (VG) methods in VQA aim to strengthen a model's reliance on question-relevant visual information. |
| Approach: | They propose to strengthen a model's reliance on question-relevant visual information by using a visual grounding method that is based on a question-related visual input. |
| Outcome: | The proposed methods can be much more effective when evaluation conditions are corrected. |
AbsVis – Benchmarking How Humans and Vision-Language Models “See” Abstract Concepts in Images (2025.emnlp-main)
Copied to clipboard
| Challenge: | Abstract concepts like mercy and peace lack clear visual grounding, and therefore challenge humans and models to provide suitable image representations. |
| Approach: | They propose a dataset of 675 images annotated with 14,175 concept–explanation attributions from humans and two Vision-Language Models where each concept is accompanied by a textual explanation. |
| Outcome: | The proposed dataset compares human and VLM attributions in terms of diversity, abstractness, and alignment, and shows that overlapping concepts are most preferred. |
Imagination and Contemplation: A Balanced Framework for Semantic-Augmented Multimodal Machine Translation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Multimodal Machine Translation (MMT) is effective in resolving linguistic ambiguities, but visual information often introduces redundancy or noise, potentially impairing translation quality. |
| Approach: | They propose a semantic-augmented framework that integrates "Imagination" and "Contemplation" they first generate synthetic images from source text and align them with authentic images via an optimal transport loss . |
| Outcome: | The proposed framework outperforms baselines on translation datasets with visually ambiguous or weakly correlated content. |
Aligned Dual Channel Graph Convolutional Network for Visual Question Answering (2020.acl-main)
Copied to clipboard
| Challenge: | Existing graph-based methods focus only on relations between objects in an image and neglect the importance of syntactic dependency relations between words. |
| Approach: | They propose a dual channel graph convolutional network to capture relations between objects in an image and syntactic dependency relations between words in a question. |
| Outcome: | The proposed model achieves comparable performance with the state-of-the-art approaches. |
Vanessa: Visual Connotation and Aesthetic Attributes Understanding Network for Multimodal Aspect-based Sentiment Analysis (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to analyze images focus on superficial features or descriptions, omitting subtle contextual information. |
| Approach: | They propose a Visual Connotation and Aesthetic Attributes Understanding Network (Vanessa) for Multimodal Aspect-based Sentiment Analysis. |
| Outcome: | The proposed network captures both implicit and explicit sentimental cues and can be used to enrich textual sentiment analysis. |
Can VLMs Recall Factual Associations From Visual References? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a systematic deficiency in the multimodal grounding of Vision Language Models is identified . VLMs can recall factual associations when provided a textual reference to an entity . |
| Approach: | They identify a systematic deficiency in the multimodal grounding of Vision Language Models . they show that VLMs struggle to link their internal knowledge of an entity with its image representation . |
| Outcome: | The study shows that VLMs struggle to link internal knowledge with image representations . the findings provide recommendations for future research . |
Enhancing Multimodal Retrieval via Complementary Information Extraction and Alignment (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies focus on capturing information in multimodal data that is similar to their paired texts, but often ignores the complementary information contained in multimodule data. |
| Approach: | They propose a multimodal retrieval approach that employs Complementary Information Extraction and Alignment to capture complementary information in multimodal data. |
| Outcome: | The proposed approach achieves significant improvements over divide-and-conquer models and universal dense retrieval models. |
Lost in Embeddings: Information Loss in Vision–Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Experiments reveal connectors substantially distort the local geometry of visual representations, with k-nearest neighbors diverging by 40–60% post-projection, correlating with degradation in retrieval performance. |
| Approach: | They propose two approaches to examine and quantify information loss by analyzing latent representation space. |
| Outcome: | The proposed model improves retrieval performance by analyzing changes in k-nearest neighbor relationships between image representations before and after projection. |